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THE DISCREPANCY EVALUATION MODEL 
An Approach to Local Program Improvement and Development 

INTRODUCTION 

Three years ago in Pittsburgh's Big City Evaluation Conference 
Report to Office of Education, we wrote, "Title I of the 1965 
Elementary and Secondary Education Act may be forgotten as a 
poverty act but long remembered as the source of systematic self- 
appraisal in America's schools. " 

The statement was not prophetic. Useful evaluation theory and 
practice are no better established in public schools today than they 
were then. However, the need persists. 

In most public school systems, evaluation consists of preemptive 
applications of qua si- experimental designs and abortive efforts to 
improve programs which were poorly designed and installed and 
remain poorly administered. 

Ultimately programs will improve only if teachers, administrators, 
and students in most of America's classrooms become involved in 
a comprehensive effort to review and improve their own work. Such 
an effort requires careful study by school staffs of their program 
operations, a detailed analysis of program inputs and processes, and 
the verification that programs are in fact operating as people believe 
them to be operating. 



For two and a half years in Pittsburgh a carefully selected 
staff has been engaged in the construction of an evaluation model. 

Our mandate was clear: redefine the purpose of evaluation in a 
manner acceptable to local, state, and federal education agencies 
and then devise and test an operational evaluation model based on 
sound theory capable of achieving these multiple purposes. 

The Office of Education has for three years been forced to go 
to the Congress with an annual ESEA report purely descriptive in 
nature- -even though the Hill has constantly asked for information 
as to program benefits. Many state education agencies have asked: 

"Is our money being spent wisely? " Local boards of education 
too have wanted to know about program benefits to children. Clearly, 
what has come to be known as a product evaluation or a benefits 
assessment is desired at all levels of government. Those who have 
attempted such evaluations at all levels have discovered what 
educational researchers have known for a long time: when quasi- 
experimental designs are applied to the outcomes of new educational 
programs, generally no evidence of new program advantages over 
existing program is obtained. 

It was with this knowledge almost three years ago that the Pittsburgh 
evaluation staff focused on a second major purpose of evaluation: to 
obtain sufficient information about the operation of new programs to 
change and improve them in their early stages of planning and installation. 
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This purpose was more relevant to local than state or federal 
evaluation needs, but obviously the long-run effectiveness of 
programs at all levels would be due to the adequacy of their manage- 
ment, their fidelity to program design, and the soundness of the 
designs themselves. Finally, because of the long development 
time of new programs, a third purpose arose: the need of local 
or state officials to make early predictions as to a program's 
eventual success or failure and to use these estimates as the basis 
for terminating high failure risk programs. 

All three of these purposes are served by the Pittsburgh Evaluation 
Model. 

Clearly, the original purpose of Title I legislation was to increase 
education program payloads for the underprivileged and to prove it. 
Before this can ever be possible we are going to have to improve our 
ability to design, install, and stabilise programs. It is important to 
note that an improvement in development is not always immediately 
reflected in performance. For example, in aviation a change in wing 
design may be ineffective until coupled with increased horsepower. 

So in education a change in instructional material may be inconsequential 
until coupled with a new mix of students or a new teaching technique. To 
estimate the effects of a change in program procedure by looking at 
product outcomes may be grossly misleading. 
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We must be able to demonstrate our ability to systematically 
improve programs which will in the long run result in educational 
benefits. Some program development work in universities, regional 
laboratories, and centers goes forward. However, it is obvious that 
unless massive aid is given to research for program development 
work, little improvement in benefits will be seen in this decade. 

All Title I and III ESEA programs are new programs in the 
sense that they are new to the school district and to most of the 
personnel involved,- These programs are in a "becoming stage" for 
staff. Procedures must change with experience. And as procedures 
change, as the possible and impossible are sorted oijt, goals change 
c*.nd the product to be evaluated must change. 

It is possible to describe America's educational task using the 
following equation: 

I(P) = O 

Where "I" = input, "P" = process, and "O" = output. Outputs are 
viewed as a function of the interaction of inputs with process. For 
example, students, teachers, and materials (inputs) interact in such 
a manner (process) as to produce a change in reading levels (output): 
The difference between "the goal" of the program and the output of the 
program CG-Oj should be minimized for program success. 

One of the major purposes of program development work is to 
better understand the relationships described by this 



equation in regard 



to any program. As more information is obtained aiid used as to just 
what inputs, processes, and outputs are involved, a program becomes 
better defined, more easily operationalized, and ultimately is likely to 
be more productive. 

"What educators must realize is that even after a program is 
initiated they must update their knowledge of all three elements of the 
equation. If, for instance., a program manager fails to define how inputs 
will change to outputs (i. e., process) he may be able to demonstrate that 
a program has achieved its purposes at a given level of cost, but he will 
be unable to say what the program was or why it worked. The success 
of any replication will depend on the administrators ability to understand 
and apply the principles underlying a program rather than to reproduce 
the specifics of that program. 

I on the other hand, he defines process well but fails to adequately 
define either inputs or outputs he. will either have created a program 
whose cost and prerequisite conditions are unknown and unreproducible 
or he will have inadequate information as to the full range of outputs of 
his well-defined program. 

In the early design stage of a program, the terms in the equation 
will normally be minimally defined. As program adjustments are made 
over time, these terms will have to be redefined at about the same level 
of specificity. The most common error of the evaluator is to sharpen 
the definition of one term without making corresponding adjustments in 



the other terms. For example, to redefine the process of a reading 
program by focusing on the quality of student and teacher interaction 
without giving careful attention to the kind of student affect which will 
be an immediate outcome of such interaction is likely to load to error. 

An example of such error would be the conclusion that a) the 
specific interaction process is ineffective, or b) the entire program is 
ineffective. Corresponding attention should be given to defining such 
inputs as teacher qualifications for such interaction. Again, if teacher 
qualifications are not carefully defined as to ability to engage in such 
interactions, the same conclusions (a and b above) may be erroneously 
drawn— whereas in fact process and program could be sound and only a 

change in input might be required to keep this program developing 
nicely. 



On the assumption that new educational programs inevitably 
cnange over time and that their design and implementation reflect such 
change, the job of the evaluator is to see that the definitions of "O, " 

M P, " and "I" are continuously revised at the same level of specificity. 
The task here is to extend the definition of one element to the point where 
an obvious lag in the definition of the other two elements is apparent. 
Program development is a function of this kind of leap-frog progress. 

The model described here deals with explicit methods for using 
evaluation as a program improvement tool as well as a means of pro- 
gram assessment. Yet the publication of this Model may outmode it. 
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The amount of effort, time, and resources needed for a school system 
to do the kind of ongoing program design work described here should 
make abundantly clear the dependence of school systems on "canned" 
programs developed by commercial or non-profit organizations such 
as the national R & D Centers. Unfortunately only a few people in the 
government and the centers themselves seem to recognize the kind of 
program development v/ork and supporting program specifications and 
standards that are necessary- before either Regional Laboratories, 

Title III Centers, or public schools can move to install and maintain 
these programs. 

The authorship of this report is lost in a welter of staff work done 

over the last three years. However, special recognition should be given 

to the work of Judith H. McBroom, Coordinator of Evaluation; Gordon A. 

Welty, Staff Training Specialist; Mary Jane Duda, Coordinator for Research 

for Instruction and Teacher Training; Leonard E. Glassner, Research 

Associate; Esther Kresh, Research Associate; and Richard Fogel, Graduate 
\ 

Assistant. 

We would also like to thank Ruth Ryals for her able editorial assistance. 
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THE MODEL 



The Discrepancy Evaluation Model described in this report is the 
result of an attempt to apply evaluation and management theory to 
the evaluation of programs in city school systems. 

There is surprisingly little theory on which to base meaningful 
evaluation practice. The references which have bfcrin most relevant 
to the work reported in this model are listed in the bibliography at 
the end of this volume. 

Daniel Stufflebeam and Egon Guba have published a number of 
private papers which make substantial contributions to the under- 
standing of institutional change and growth and provide a theoretical 
frame of reference for the assessment of change. However, despite 
the name of an education publication at Ohio State University, Theory 
In to Practice , there appears tc be very little linkage between program 
evaluation going on in public schools today and the kind of theory under 
discussion by a few university theorists. 

Assumptions 

1. Many educational programs, especially federally-funded programs, 
are installed in public school systems without adequate planning. 
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2. Given this fact, evaluation should be a process for program 
development and stabilization, as well as a means of assessment. 

Tu accomplish this purpose, evaluation must provide information 
which decision makers can use to improve, stabilize, and assess 
programs. 

3. Two decision-making audiences exist for program evaluation 
information. Those responsible for making decisions to improve 
and stabilize specific programs are the first and primary audience. 
This audience is composed of all strata of program staff- -from 
paraprofessionals up through the top program administrator. Those 
responsible for making decisions to retain or terminate various 
educational programs, that is, decisions relative to the allocation 
of resources, are the ultimate audience for evaluation information. 
This audience is at the policy-making level of an entire school 
system. 

4. The involvement of program staff in the process of evaluation fosters 
commitment to program improvement and a more analytical approach 
to the program which results in desired changes in staff behavior. 

5. Evaluation and decision making are separate, yet complementary, 
functions. Therefore, program evaluators must maintain their 
independence of program staff and at the same time assume a non- 
directive role. 



The Stages of Evaluation 

Evaluation at its simplest level may be seen as the comparison of 
performance against a standard. When evaluation is viewed as a process 
for program development, stabilization, and assessment, as is the case in 
the Discrepancy Evaluation Model, there are five such relevant com- 
parisons. For convenience, the comparison of each level of performance 
with an appropriate standard designates an evaluation stage. These 
relationships are shown in Figure 1 below. 



Evaluation Stages 



Stage 


Pe r f o r m an c e 


Standard 


I 


Program Design 
Input Dimension 
Process Dimension 
Output Dimension 


Design Criteria 


II 


Program Operation 


Program Design 
Input Dimension 
Process Dimension 


III 


Program Interim Products 


Program Design 
Process Dimension 
Output Dimension 


IV 


Program Terminal Products 


Program Design 
Output Dimension 


V 


Program Cost 


Cost of Other Programs 
with Same Product 



Figure 1 



At all stages, some indicator of performance is obtained which is 
compared with a standard whicii serves as the criterion of performance. 

At Stage I, a description of the program's design is obtained as 
"performance" information. This performance is compared v/ith the 
"Design Criteria" postulated as a standard. Discrepancy between 
performance and standard is reported to those responsible for the 
management of the program* At Stage II the standard for comparison 
is the program design arrived at in Stage I. Program performance 



information consists of observations from the field regarding the 



program^ installation. Discrepancy information may be used by the 
program manager to redefine the program or change installation pro- 
cedures. At Stage III the standard is that part of the program 
design which describes the relationship between program processes 
and interim products. Discrepancy information is used either to re- 
define process and relationship of process to interim product or to 
better control the process being used in the field. At Stage IV the 
standard is that part of the program design which refers to terminal 
objectives. Program performance information consists of criterion 
measures used to estimate the terminal effects of the project. At this 
point in time, if decision makers have more than one project with 
similar outcomes available to them for analysis, they may elect to do a 
cost-benefit analysis to determine program efficiency. The "Design 
Criteria" constitutes a basic assumption on which all other criteria for 
standards are based. The "Design Criteria" has been defined so as to 
contain three basic elements, each of which may subsume many variables. 
These basic elements of any program (as described in a vast "systems" 



literature) are Input, Process, and Output. 

If an evaluation staff is to have the support of the program staff 
it seeks to evaluate, it must provide visible assistance to the staff 
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of that organization effecting change. Such assistance muss be in a 
form acceptable to program staff. The only assurance of such accept- 
ability is that program purposes be defined by the program staff and 
the methods of change be determined by them as well. There must be 

mavirmim J. _ /* * . - 

wi jjiugrcuju sian m every step ot tiie evaluation 



process, lurther, it follows that there must be continual rapport be- 
tween program staff and evaluation staff, fostered at the initiative of the 
evaluation staff and resulting in a continuous communication of affect as 
well as publicly acceptable verbalizations. The relationships to which 
an evaluation unit submits itself are binding and pervasive? however, it 
does not follow that evaluation therefore operates at the discretion of 



the administrator of the program unit. Evaluation is the handmaiden of 
program development and quiet counselor to administrators --but it 
operates in accordance with its own rules and on an authority independent 
of the program unit. 

An organizational paradigm which makes these intricate and 
demanding relationships understandable is that of an action system 
which contains a feedback loop. The processing of input is at the dis- 
cretion of the program unit. The definition of output and the shaping 
of input are at the discretion of the parent organization. The management 
of the feedback loop is in the hands of evaluation staff. The feedback 
consists of discrepancy between performance and standard. There 
can be no evaluation without discrepancy information. There can be 
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no discrepancy without a standard; therefore, the first task of any 
evaluation is to obtain program standards. 

A feedback loop of discrepancy information based on standards 
derived from the program staff will necessarily be of interest to a 
program staff which has been given responsibility for the success of 
its program. 

The evaluation of any school program, which is already staffed 
and underway, goes through four major developmental stages of 
comparison — each of which may deal with input, process, and output. 

This process of comparison over stages takes the form of the 
flow chart in Figure 2. 

C/B Analysis 




Figure 2 



In the chart S - standard, P = pregram performance, C = compare, 
D = discrepancy information, and A = change in program performance 
or standards. Stage V represents a cost-benefit option available 
to the evaluator only after the first four stages have been negotiated. 

i 

i 

Notice that the use of discrepancy information alv/ays leads to a decision 
to either 1) go on to the next stage, 2) recycle the stage after there has 
been a change in the program's standards or operations, 3) recycle to 
the first stage, or 4) terminate the project. From the program 
manager's point of view, discrepancy information permits him to 
pinpoint a shortcoming in the program for one of two purposes: to 
change the operation of the program, or to change the specifications 
under which the program operates. A superintendent of schools or 
board of education will be as concerned with the rate of movement of 
a project through its evaluation stages as with discrepancy information 
at any given stage. Generally, the longer it takes to get to stages two, 
three, and four, the greater the cost if the project fails. The faster 
a project moves into advanced stages, generally the less the risk of its 
failure. 

Various kinds of performance and standards and the implementation 
of comparisons will be discussed in greater detail under each stage. 



- 14 - 




STAGE I 



At Stage I, the performance to be assessed is the program design* 
Experience suggests, however, that with ESEA Title I programs there 
are usually at least three designs of the program in existence: one is 
the funding proposal, another is that held by program administrators, 
and at least one other exists in the minds of program practitioners. 

The question is raised, then, as to which program design to assess. 

. This question is settled under the model by rejecting inadequate 
designs and by generating anew dynamic baseline design by means of 
a group interview with the program staff. This interview takes 
place at the design meeting. 

If the program staff is not too large, all members may be invited 
to attend the meeting. If it is quite large, a sample of each job type 
may be selected. The entire group may be divided into smaller 
groups on the basis of staff level to avoid status conflicts that would 
inhibit discussion. Or, the groups may contain representatives 
from all levels. The evaluator structures the meeting to fit the 
individual program. Whatever the case, all design meetings have 
these two common ingredients: all levels of program staff are 
represented, and the larger group is broken down into smaller groups 

to facilitate discussion. 
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The purpose of the design meeting is to obtain information which 
will ultimately become the program design. Prior to the meeting a 
set of very specific questions is drawn up by the evaluator of the 
program to elicit this information. These questions are derived from 
the "Program Design Criteria 11 shown in Figure 3. 

The "Design Criteria" includes a comprehensive list of program 
elements. An educational program is viewed as a dynamic input-output 
system with specifications for inputs, process, and output being necessary 
and sufficient for program design. 

By examining Figure 3, it can be seen that every program must 
specify the variables it seeks to change. These variables will have 
limits or values set on them to specify the entering levels of students 
under inputs and another set of values to specify the goals of the program 
under outputs. For each pair of change variables (that is, for each 
input-output pair) there is a process to transform the value of the input 
dimension to the desired value of the output dimension. In defining 
the process it is necessary to find conditions sufficient to effect 
this change. 

In addition to the change variables, there are preconditions for 
each program. These designate the resources prerequisite for, but 
unchanged by, program operation in terms of students, staff, adminis- 
trative support, facilities, media, and time. As in the case of the 
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DESIGN CRITERIA 



Inputs 



“variables — ths things ths pro= 



gram is attempting 
to change 



A. 

B. 

C. 



Student Variables 
Staff Variables 
Other Variables 



E. Preconditions --the things that 

are prerequi- 
site to program 
operation yet 
remain cons tan 
throughout the 
program 

Student Conditions 
Staff Qualifications 
Administrative Support 
Media 
Facilities 
Time 

III. Criteria must be specified for 
each input variable and pre- 
condition above. The criteria 
specified for student variables 
and preconditions constitute 
the selection criteria of the 
program. 



Process 

Variables --those activities which 
change inputs into de- 
sired outputs 

A. Student Activities 

B. Staff Activities 

1. Functions and Duties 

2. Communication 

a. Intra- staff 

b. With Others 



Criteria must be specified for 
each of the process variables. 



Outputs 

Variables --the changes that have 
come about 



A. 

B. 

C. 



Student Variables 
Staff Variables 
Other Variables 



Preconditions — same throughout 
the program 



Criteria are specified on the 
variables to define the goals of 
the program. The participant 
is released from the program 
if he achieves -:he goal of the 
program or if he violates a 
precondition. 



Figure 3 
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variables a limit or level is set on each precondition. The combination 
of the student preconditions and the student variables constitutes the 
selection criteria of the program. The goals of the program, however, 
are defined only in terms of the variables. Thus, a student may be 
released from the program either when he achieves the goal(s), which 
would constitute a successful completion, or without achieving the goal 
but by violating a precondition such as the completion of a specific 

grade level, which would constitute an unsuccessful completion. 

Aside from the purpose of gathering information to satisfy design 
criteria, the program design meeting serves another important function- - 
that of consensus building. Through the process of give and take that 
occurs in a discussion group, program staff come to some agreement 
about their purposes and procedures. In the course of reaching 
consensus, strong opinions are promulgated and contested, forcing 
the discussants to think more analytically and carefully about their 
program and fostering a commitment to the program. The consensus 
which is generated constitutes the authority for use of the program design 
as a standard in Stage II. 

Once a design has been derived from program staff, activity is 
channeled toward making the Stage I comparison. The program design 
(performance) is assessed for comprehensiveness and internal consistency 
against the "Design Criteria" (standard). The vehicle for conducting the 
Stage I comparison is a panel meeting. 



The panel is composed of those persons who are most involved 
in and knowledgeable about the program: the program administrator, 
the program evaluator, a consultant who is an expert in the area of the 
program content, the Coordinator of Evaluation, and a resource person 
from the Office of Research who has a background in program design. 

The panel is the mechanism used to preserve the judgmental function 
of evaluation and at the same time to reduce the possibility of error 

by using expert opinion in the formulation of those judgments. 

In assessing the adequacy of a program design, there are two basic 

questions relative to the criterion of comprehensiveness: 1) "Is there 
specific and complete information for each element of the program design? " 
2) "Is the information in useable form?" That is, "Is there an adequate 
criterion for each of the variables and preconditions?" The answers to 
these questions will depend on the adjudged face validity of the program 
design relative to the "Design Criteria. " In the case of a disagreement 
among panel members, the evaluator is responsible for final judgment. 

The consultant, on the other hand, has the final authority in 
determining the internal consistency of a program design. Questions 
of internal consistency will revolve around the relationships between 
the various elements in the design. For example, "Are student 
activities sufficient to change the variables from their input state to 
the defined output state? " "Is sufficient time allotted to program 
activities to be able to achieve program goals? " and "Are staff 
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qualifications sufficient to enable the staff to perform their functions 
and duties? 11 In short, internal consistency lias to do with the soundness 
of the design in relating theory to practice. A complete list of questions 
to bo asked in determining the internal consistency of the design is 
included as Figure 4. 

In addition to assessing the program design on the criteria of 
comprehensiveness and internal consistency at Stage I, the panel 
also investigates the external consistency of the program. This involves 
a study of the compatibility of the program with other programs operative 
in the entire school system. It is essential that programs in conflict 
with each other be identified. In the absence of explicitly defined 
system values, the opinions of major staff- - both members of the given 
program and those who have a more comprehensive view of the system, 
such as the principal “"are solicited. Although judgments arrived at 
in this manner are indeed gross, it is important to have information 
about possible value conflicts as to the use of student or staff time, 
facilities, and media. Problems relating to compatibility are difficult 
to solve since they often require the ranking of values for the entire 
system. If any such obstacle to program success is present, it is 
important to identify it at the beginning. 



Checklist for Internal Consistency 



Yes No 

Staff qualifications are sufficient for performing 

staff functions and duties. 



2. Staff duties are clearly related to staff functions. 

3. The administrative support is sufficient for 
program operation. 

4. Media are related to and sufficient for student 
activities. 



5« Facilities are adequate for program operation. 

6. The time allotted for program operation is 
sufficient to accomplish program goals. 

7. At least one of the student variables is a 
selection criterion. 



8. Student activities are related to student goals. 

9. Staff activities are related to student goals. 

10. A process is defined that is sufficient to change 
each input variable into the output variable. 

11. Communication activities within the program and 
between the program staff and others are sufficient 
to support operation. 



Figure 4 
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The evaluator conducts field interviews using Basic Interview 
Questions for Program Compatibility shown in Figure 5. He presents 
the findings of these interviews at the panel meeting for review, and 
is the ultimate authority in making judgments relative to compatibility. 

In summary, the purpose of evaluation at Stage I is to derive a 
design of the program and to assess that design according to its com- 
prehensiveness and internal consistency. In addition, the compatibility 
of the program within the school system is determined. The standard 
for making Stage I judgments is the "Design Criteria. " The first design 
of a program is likely to lack specificity and internal consistency, but 
Stage I procedures provide a mechanism for making it ever more 

refined and sound. 

‘ * 
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Basic Interview Questions for Program Compatibility 



1. Is sufficient time available for student participation in the program? 
What activity does the student give up in order to participate in 

the program? 

Does this reallocation of student time result in sacrifice to other 
objectives of the school program? 

Does it have an effect on the operation and/or goal attainment of 
this program? 

2. Is sufficient time available for participation by the program staff 
and cooperating personnel? 

What activities do staff or cooperating non-program personnel 
sacrifice in order to participate in the program? 

Does this reallocation of their time result in a sacrifice to other 
objectives of the school program? How does it affect this 
program? 

3. Are facilities and media now available to the program? If not, 
why are they absent? 

Is allocation of facilities and/or media to this program resulting 
in sacrifice of other objectives of the school program? 

Is this program affected by the manner in which facilities and 
media are allocated? 

4. Are the gains for students anticipated by this programs equal to, 
less than, or greater than possible sacrifices in other educational 
objectives of the school program? 



Figure 5 
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STAGE H 



At Stage II the design, which was the performance at Stage I, 
becomes the standard against which to judge the program operation. 

In making the comparison between program operation (performance) 
and program design (standard), the evaluator proceeds item by- 
item through the program design, considering each item for a 
congruence test. All statements in the program design are subject 
to a comparison with what is going on in the field. However, the 
evaluator bases his decision of which variables to test at a given 
time on considerations of convenience and knowledge he has gained 
from the panel meeting as to the possible inoperability of certain 
design elements. This selectivity is introduced because of the 
AAiiiiLed resources available to the evaluator. A tradeoff at any given 
time is effected between those aspects of the program easiest to 
investigate a,nd those most likely to evaporate. Eventually the 
evaluator will submit all elements of the design to a congruence test-. 

Ihe congruence test is facilitated through a series of observations — 
some of which may require indirect measurement. All of the usual 
problems of measurement may pertain here and standard references 
and techniques are used by the evaluation staff. However, the wise 
evaluator limits his first round of congruence testing to easily 
understood referents which have high credibility for the program manager. 
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In practice, it has been found that a congruence test of "input" elements 
can and should generally precede a test of "process" elements. After 

discrepancy information on "input" has been formalized and feedback 
given to the program manager, a new cycle of "process" congruence 
testing can be initiated. This latter test is characterized by verifica- 
tion of the existence of process elements. It is not concerned with 
verifying whether the relationship between input and output due to pro- 
cess does in fact exist. Such a study of causation is dealt with in 
Stage ID- 

After each series of congruence tests, the evaluator provides the 

program manager with the information obtained. If there are 
discrepancies, the program manager has only two options: he can 
either modify the design of the program or modify the program operation. 
We can thus see how the program manager, on the basis of information 
provided, proceeds to equalize program operation and design. 

The question as to the criteria to be used or how much discrepancy 

is to be considered as inevitable has not yet been resolved under the 

model. In the absence of a criterion, the procedure used is a group 

« 

judgment of the same type obtained in Stage I. The panel meeting is 
called by the evaluator when a first set of elements of the program design 
has been subjected to a congruence test. Again the consultant brings 
his theoretical expertise to bear on the subject, and the evaluator 




draws on his knowledge of the program design. In addition, where 
indirect measures have been used the psychometrician is responsible 
for the construction of instruments, and comments on the validity and 
reliability of findings. The questions to be answered by the panel are: 
1} "Is the information on each program element complete? " 2 ) "Is 
it reliable and valid? " 3) "Are the discrepancies uncovered ones which 
will significantly diminish the program's chances for success? " The 
evaluator is the ultimate authority in answering the first question, 
the psychometrician the second, and the consultant the third. 

Through Stage II work the congruence between the program 
design and operation is continually increased. Such an increase in 
congruence is what is meant in this model by program improvement. 
When the panel decides the program is sufficiently stable, a decision 
is made to move to Stage III. 



STAGES III and IV 



At Stage ILL, the first cause and effect comparison is made. The 
relationship tested is between the variable to be changed (from input to 
output) and the process or treatment used to effect this change. This 
relationship, as predicted in the design, is the "standard. " The relation- 
ship found to obtain empirically is the "performance. " Another way of 
stating this is, does "P" change "I" into "O"? This question is asked 
in two sequential stages: at the microlevel of specific program process 



activities producing specific enabling outcomes (Stage in) and at the 

macro level of the entire program or gross treatment producing the 
anticipated outcomes of the total program (Stage IV). Stage IV adheres 

to the traditional use of experimental and quasi- experimental designs 



and is discussed only briefly here. 

The purpose of Stage III evaluation is to provide program staff 
with an estimate of the effect of the process elements (or treatment 
variables) on the output elements (or dependent variables) as a function 
of time. In order to accomplish this goal, continuous measurements 
must be taken both of treatment variables and dependent variables. 
These assessments result in graphs, which taken together provide a 
useful description of the process effects on the dependent variables. In 
order to assess interaction effects, these graphs may be kept for each 
class and each school in which the program is installed. In addition, 
a graph should be kept on total program population. The frequency of 



the measurements will vary from program to program and will generally 
be determined by the program staff's assessment of the minimum unit 
of time it takes to produce a measurable change in the dependent 
variable. If the dependent variable is achievement in a math text, this 
unit of time might conceivably be defined as a week's work. If the 
dependent variable is change in attitude, a measurement might be 
taken every few months. 

« .jdl Stage in the initial effects of partial treatment are assessed, 
further adjustments in treatment are made based on an analysis of 
interim product data, and greater understanding is achieved of the 
relationship between treatment outcomes and the conditions of the 
experiment. At Stage III, the evaluation staff should collect data 
describing the extent to which student behavior is changing as predicted. 
Most of this activity is microscopic in nature. Such evaluation depends 
heavily on the production and use of highly specific instruments that 
provide empirically determined answers to cause and effect questions. 

As a consequence of this stage of evaluation, the program staff learns 
whether or not its intermediate program payloads are being realized 
on target dates, and if not, why not. 

Consider the following example: 

A program is installed to improve reading achievement. The 
primary treatment is an individual tutorial program in which the 
teacher works with one student at a time while the other students are 
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engaged in various individual assignments. Suppose, for example, as 
often happens, a pretest and posttest are administered. The results 
indicate that the students in this program improve their reading ability- 
no more than similar students in previous years who were not involved 
in the program. A traditional report would indicate no differences, and 
program staff would be uninformed as to reasons for the failure. Now 
consider what could occur under a system of continuous assessment of 

treatement and dependent variables. 

Figures 6 and 7 represent some rather exaggerated but neverthele 
possible examples of what the line might have looked like if continuous 
assessment had been used. 

Each time period represents a measurement. 

Achievement Plot — Restricted Gro-wth 




Pretest Posttest 



Figure 6 



Achievement Plot--Gradusi Growth 
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2 3 



4 5 6 

Posttest 



Figure 7 

Given these two plots, it would be possible at point C, or at least D, 
to inform program staff that the amount of time spent in individual instruc- 
tion has fallen off and no growth in reading is in evidence. Program staff 
could step in at this point and either change the design of the program or 
investigate the existence of new variables to be operationally controlled or 
stabilized. 

Finally, at Stage IV the evaluator may cast an experimental design 
which answers the question: "Has the program achieved its major 

objectives? " 

Stage IV calls for the kind of designs we have long employed in educa- 
tional research and have more recently employed in error in evaluation. 
"Employed in error, " not because the quasi-experimental designs of the 
type described by Stanley and Campbell do not belong in an evaluation 
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strategy, but because they have consistently been used in the wrong stage 
of a program's development. 

There are conditions prerequisite to the use of experimental design 
in a school setting, and one of the purposes of the early stages of an 
evaluation is to secure these conditions --just as the early stages of 
program development form the base on which later program growth may 
be realized. 

In Stage IV, many of the relationships between treatment conditions 
and effects discovered in Stage III can be properly expressed as independent 
variables in the experimental design stage. The administrative control 
secured over the new program in Stages II and III ensures treatment 
stability. Problems of experimental design, sampling, and instrumentation 
are more likely to be solved because of increased staff knowledge of 
factors interacting with treatment. 



STAGE V 



Having completed evaluations at each stage of the program* s develop- 
ment, it is possible to conduct a cost-benefit analysis of the entire program 
and to compare the results of that analysis (performance) with similar 
cost analyses of other educational programs designed to achieve similar 
results (standard). The objective of such comparisons is to determine 
the most effective allocation of resources. The key to achieving a 
functional cost-benefit analysis is the extent to which inputs and outputs 
can be given measurable costs and benefits. If objectives and elements 
of the design criteria of various programs are given similar quantifiable 
classifications initially, comparison of cost-benefit analyses is useful. 

Cost-benefit analysis normally depends on the establishment of a 
curve which is a function of benefits relative to costs. Such a curve 
permits one to relate increments of benefit to cost. The development of 

such curves goes beyond this model. 

It is early in the development of cost-benefit analysis in the public 
sector to be specific about the procedures to be employed. In fact, at this 
time, information is simply notavaiiable for comparisons across programs. 
However, cost-benefit analysis is the ultimate rational step in the process 
of program development and assessment put forth in the Discrepancy Modei. 
In anticipation of its eventual use, the cost-benefit analysis is posited as 



StageV. 



The Dynamics of Evaluation 



Although for purposes of explication the stages of evaluation have 
been presented abpve as if they were self-contained and sequential, the 
real nature of evaluation work is dynamic, with much overlapping and 
interplay between the stages. This is so because evaluation not only 
stimulates program development but also must be conducted in light 
of program change. What occurs, therefore, is evaluation, program 
change, and reevaluation, allowing sufficient time between evaluations 
for program change to take place. 

In actual practice it turns out that an evaluation requires frequent 
recyclin'* through those stages which are prior to the stage under nego- 
tiation at any point in time. Successive reappraisal of program operations 
and the program design from which program operations are derived is 
generally a consequence of the decisions made by program staff on the 
basis of discrepancy information reported at Stages II, III, and IV. If 
a decision is made to reformulate program design rather than to revise 
program performance, there are immediate implications for the rene- 
gotiation of all subsequent evaluation stages. Hence, the soundness of 
judgment of program decision makers and the support they derive from 
their organizational milieu are of prime Importance to evaluators. 

This is particularly true of evaluation activity in Stages I and II 
where major program changes can be expected to occur. When a program 
is first defined, the design is generally neither comprehensive nor 
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internally consistent. The panel meeting is then held and problem areas 
in the design are identified. It is sometimes possible at a panel meeting 
to resolve difficulties or to fill in gaps from information provided by the 
program administrator. If so, some changes in the design can be incor- 
porated but a large number of questions remain to be recorded and 
brought to the attention of the entire program staff at a later date. 

Vague and inconsistent as the first design is, there are usually some 
elements which are specific and complete enough to provide a standard 
for Stage II congruence testing. After statements in the design have 
been compared to program operation, the program administrator is 
supplied with the discrepancy information. (Experiment has shown that 
at this early stage in program development, discrepancies almost always 
exist. ) He then has two alternatives: to change the design of the program 
or, as is more usually the case, to adjust program operation through 
communication of program intent to the practitioners in the field. Whichever 
the case, some change occurs which, after the time lag for implementation, 
must then be evaluated. 

At this point a second design meeting is often held to obtain needed 
additional information, to refine existing statements which are not specific, 
or to restore internal consistency after changes have been made in one 
section of the design as a result of previous evaluation activity. For 
example, if the panel meeting made a change in the statement of goals, 
process would then have to be adjusted to make it consistent with goals. 
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The second design is usually far more developed than the first. 
However, if, in the opinion of the evaluator, problems still exist, a second 
panel meeting may be called. As changes in design occur, more Stage I 
work is done, program operation is amended, and the process repeats 
itself until the decision is made {in a Stage II panel meeting) that the pro- 
gram is sufficiently stable to measure initial outputs. Even while Stage HI 
work is being undertaken program operation is being monitored to maintain 
its accord with program design. Thus, it is necessary to emphasize that 
not only is the movement through stages not sequential, it is also not linear. 
The stages of evaluation activity may more accurately be said to occur 

concurrently and can be represented as shown in Figure 8. 

INTER-STAGE RELATIONSHIPS 




Figure 8 



Stage I activity is represented by the largest circle: Stage II then 
occurs within the context of Stage I work; Stage III work is accompanied 
by the monitoring of Stages I and II; Stage IV is accompanied by the moni- 
toring of Stages I, II and III; and Stage V work utilizes the findings of 
Stage IV work for the specific program while overlapping onto the Stage IV 
work of other program evaluations. 

Thus, we see that the process of evaluation is organic and dynamic 
and grows not only out of previous evaluation findings but out of program 
change as well. It is a long and complex process, but when the benefits 
of a stable program can be assessed and then compared to the benefits of 
other such programs with the aim of determining the most efficient means 
of reaching an educational goal, the preliminary travail should prove to 
be worthwhile. The real payload of this elaborate approach is the increased 
probability of improved programs. 

At a more practical operational level, evaluation activity, which is 
by its nature continuous, is divided into manageable segments called cycles. 
The cycle begins with a plan which includes a group of questions immediately 
relevant to the demands of the evaluation of a particular program and sets 
out the procedures to be used in answering those questions. After a cycle 
of evaluation has been conducted which includes the entirety of Stage I, any 
additional cycles will include questions pertaining to more than one stage. 

At the completion of a cycle, a cycle report is discussed with the program 
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administrator. This verifies the requirement that evaluation provide 
information for decision making. 

Information for Decision Maker s 

As stated in the assumptions, those responsible for making de- 
cisions about one or more programs are the first and primary audience 
for evaluation information. The types of feedback given may be 

characterized as formal or informal. 

After each cycle of evaluation activity, members of the program 
staff are provided with formal feedback in the form of a cycle report. 

The report contains information about the problems in program design 
or operation which require adjustments. The evaluation function is not 
seen as providing a solution or alternate solutions to the problems, but 
rather as presenting problems based on discrepancy information. The 
evaluation model represents a method for dealing v/ith these problems. 

The cycle report is always given to and discussed with the program 
administrator, who has the courtesy of a preview of each written report 
before it is issued. Since it is assumed that all strata of program staff 
have some decision-making powers, the criterion used in further distribu 
ting reports is whether a given group or level of staff can make decisions 
to affect program change on the basis of given information. Thus, if the 
findings concern only teacher activities, the report is distributed to 
teachers, but not to paraprofessionals. In addition, reports are 
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distributed to adults in the system, regardless of whether they are 
members of the program staff, who have contributed to evaluation findings. 

Feedback of the informal type is provided to program staff by the 
evaluator who interacts almost continuously with both administrative 
and field personnel. The degree of interaction is determined both by 
the size and scope of the field to be covered and by the number of 
scheduled activities. Program activities such as in-service training 
meetings and group planning sessions, as well as scheduled evaluation 
activities, provide opportunities for informal contacts. The evaluator 
seizes every opportunity for communicating recent evaluation findings. 

The timeliness of feedback is important. Thus, it is provided as promptly 
after each set of evaluation activities as is consistent with care and 
accuracy of data handling and may be presented in oral form while 
written reports are in preparation. 

The crucial factor in a program's ultimate chances for success 
is the receptivity of the program administrator to evaluation information. 
The first evaluation efforts with a program may produce a vague design 
and serious discrepancies between design and operation. However, given 
the pro , ram manager's cooperation and sufficient time, these problems 
can be solved. On the other hand, another program may have a much 
superior first design and fewer discrepancies between design and operation. 
In either case, if the program manager is not receptive to information 



provided, the program will not mature to the point where a product eval- 
uation is tenable. 

The program administrator's ability to utilize evaluation information 
is a second aspect of the program's chances for success. It may be that 
the administrator is not sufficiently analytical to devise solutions to 
problems identified in findings or that he is simply overwhelmed by them. 
Or. it may be that the source of the problem is simply beyond the admin- 
istrator’s control and is a problem of the system. This is the case with 
insufficient budget allocations or variation in program implementation 
due to conditions in the schools. 

The ultimate audience for evaluation information is at the policy- 
making level of an entire school system. Although providing information 
to this audience may appear to violate the relationship established with 
the first audience, the fact remains that policy makers need evaluation 
information in order to make rational decisions to retain or terminate 

programs and to allocate resources among them. 

As a great deal of time is required to implement all of the evaluation 

stages described here, policy makers of the school system will ask for 
information relevant to a program’s Chances for success prior to the 
completion of the evaluation. They will want indications of risk before 
the program product has he.en measured and a cost-benefit analysis 
performed. The provision by the evaluation staff of such information may 
irreparably damage cooperative staff relations painstakingly built by the 
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evaluation staff. However, the decision to jeopardize evaluation staff 
work must be at the discretion of the chief school administrator. 

Generally, information on program risk _is_ requested by policy 
makers. This information is provided to the policy makers by the eval- 
uation staff through use of the Program Interim Assessment Profile 
shown in Figure 9. This profile provides information as to the program 
administrator's amenability to program improvement and receptivity to 
discrepancy information. This amenability to improvement is measured 
by the number of changes in the program relative to the number of evalua- 
tion reports he has received. 

The Program Interim Assessment Profile is based on seven criteria 
of program adequacy. Three of these criteria--"Comprehensiveness, " 
"Compatibility," and "Internal Consistency"--relate to the adequacy of 
program design and have already been discussed at length under Stage I 
of the Model. "Program Implementation" is a summarization of all 
Stage II discrepancy information, and "Relation of Process to Outcomes" 
is a summarization of all Stage III discrepancy information. The last 
two criteria are not based on evaluation staff work. "Program Effective- 
ness" information deals with the adequacy or importance of program 
outputs in terms of the changing goals and values of a school system, and 
"Program Efficiency" asks whether the purposes of the program are of 
sufficient value to warrant the use of resources identified under the design 
of program as necessary to achieve program outputs. 
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An interim assessment of program is achieved by substituting in- 
formation needed to satisfy Program Interim Assessment Criteria called 
for in factors 1 to 6 into a risk of failure index equation shown at the 
bottom of Figure 9. The profile provided in Column "C" and the informa- 
tion or lack of information in Column "A, " as well as the response to 
factor 7, are considered by the decision maker in forming a judgment as 
to whether a program should be continued or terminated. 

The formula calls for a comparison of program performance with 
the number of times information has been reported for each factor in 
the profile (C-B). Factor 6, program efficiency, calls for some kind of 
determination of cost relative to the value of service being rendered. Some 
index ffimber of value of service ("S") is used such that where M S n is equal 
to "C" (cost) then benefits are of appropriate value. If "C" exceeds M S M 
then a negative term is added to the equation. The risk of failure index 
should not be interpreted too literally. The formula is merely a conven- 
ient way of comparing and reviewing pertinent information relative to each 
program, in a system. 

One other important aspect of providing information to decision 
makers is communication. All audiences entitled to evaluation information 
are relatively unfamiliar with the terms and concepts of evaluation work of 
this type. Although evaluation activity such as design meetings and in- 
formal contacts provides a kind of in-service training for staff, the effort 
must be made to communicate with program, staff at their present level 



of understanding and sophistication. It is encumbent on the evaluation 
staff to do this successfully. Evaluation findings must, therefore, be 
presented as concisely and clearly as possible— concisely to ensure that 
they are read and clearly to ensure that they are understood. Althougn 
the time-consuming task of searching for the right word for the right 
place may be a source of frustration to the evaluation unit, it is impera- 
tive. Evaluation findings not read and not understood are not used. 
Technical information is generaUy reported in state and federal annual 

reports. 

Staffing for Evaluation 

Evaluation of the type described above requires the following cate- 
gories of staff: 

x. Administrator (capable of maintaining high quality in research 
activities) 

2. Evaluator 

3. Editor 

4. Secretary 

5. Data handler 

6. Consultants in subject areas of programs being evaluated 

7. Consultants in instrument development and research design 

8. Data processing specialists 

In the list above, the staff mentioned in numbers 1 through 5 are 
regular members of the evaluation unit. The consultants and the data 



processing specialists are necessary adjuncts to, but not part of, the 
evaluation unit. 

Thus, it can be seen that the evaluators are non- technicians who 
have access to specialists or experts as they are needed. The person 
responsible for the quality control of the evaluation activity acts as a 
liaison between evaluators and consultants. 

In Pittsburgh, evaluators come from a wide variety of disciplines: 
anthropology, economics, education, English, mathematics, psychology, 
sociology, and even the ministry. Almost all have graduate degrees. 

Since no one person possesses all the skills needed to conduct this type 
of evaluation work, the two most important factors in considering 
applicants are flexibility and the ability to do analytical or critical 
thinking. It is felt that flexibility will permit the evaluator to adapt to the 
varied demands of the job and that his ability to do analytical thinking 
will be an asset in understanding and implementing the model. In-service 
training is provided throughout the year as the staff needs it. * 

The job of the evaluator is varied, but may be described as containing 
the following functions: planning, quantifying behavior, data collection 
and analysis, report writing, and small group leadership. 

Under the planning function, the evaluator is administratively respon- 
sible for the evaluation activities conducted with respect to his assigned 

A compilation of evaluation training documents is available from 
the Office of Research, Pittsburgh Board of Education. 
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program or programs. The fewer the programs he is responsible for, the 
more work gets done in each and the possibility of frequent feedback is 
increased. From experience in Pittsburgh it has been found that an 
evaluator can efficiently handle only one or two programs at a time. 

The evaluator plans and executes each cycle of evaluation activity 
for his program(s). It is helpful, in this respect, if he knows something 
about problem analysis and has command cf various management techniques 
such as flow charting, block diagramming, and PERT. 

As a quantifier of behavior, the evaluator should be able to design 
simple instruments such as questionnaires, interview schedules, obser- 
vation schedules, scales, and checklists. Once instruments have 

been devised, the evaluator must not only employ accepted techniques 
in collecting the data but must also conform to the conventions established 
for entrance to and work in schools. The evalua.tor should be conversant 
with, though not necesf rily an expert in, methods of data handling and 
analysis. 

The evaluator needs to understand group process and techniques 
of group leadership. He must also be able to relate to program admin- 
istrators and field personnel for exchange of information and interpreta- 
tion of findings. The relationship with the program administrator is 
crucial to the acceptance and use of evaluation information. 

Under the reporting function the evaluator must be able to write 
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reports in standard research format and terminology (for federal require- 
ments) and to write reports in lay language and format (for the two local 

audiences). Although he may have the assistance of an editor in the latter 
task, it is primarily the responsibility of the evaluator to assess the level 
of understanding of his program staff. 

Finally, the evaluation unit works as a team. Not only does the 
evaluator have the administrative responsibility for the evaluation 
activities in his program(s), he also assists other evaluators in conducting 
design meetings, data collection, and other activities. The evaluation 
unit meets at least once a week to conduct routine business, to critique 
each others work, for in-service training, or to bring the collective 
wisdom of the group to bear on particular problems. A spirit of openness 

and candor obtains as the group shares learning experiences and work 
problems. 
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A CASE HISTORY 



Standard Speech Development Program 

The Standard Speech Develooment Program, formerly called the 
Pattern Drills Program, was originally designed for seventh- and eighth- 
grade students in poverty neighborhoods in Pittsburgh. It now serves 
approximately 5, 000 students in 37 qualifying schools, and is taught 
by 92 teachers as an integral part of the regular English curriculum. 

The recommended procedure calls for a 10- to 15-minute daily drill 
on a particular phonetic or grammatical structure of standard spoken 
English. In the program the use of non-standard speech is in no way 
discredited. On the contrary, its preference is conceded for many 
daily situations in the students’ lives.. However, it is realized that 
success in middle class social and business activities requires control 
of that standard language which serves as the currency of communication 
in the larger world. Pattern drills were therefore developed to equip 
adolescents with the phonetic and grammatical structures which will enable 
them to achieve this control. With this rationale the program qualified 
for federal funding under Title I of the Elementary and Secondary Educa- 
tion Act of 1965. Initial planning began in that year. The evaluation 
staff of the Office of Research did not become involved in the program 
until it was already underway. Once this involvement had become a fact, 
program and evaluation activity began to influence each other. At this 



point the interaction between curriculum and evaluation d*- velopment 
which is detailed in this report began. Figure 10 shows the sequential 
progress and interrelationships of program and evaluation development 
through the various stages prescribed by the evaluation model. 

The first step in evaluating the program was to determine whether 
<a design existed in accordance with the design criteria. The evaluation 
scaff assumed responsibility for seeing that this was done. The evaluator 
prepared a set of questions to be used as the basis for a program definition 
meeting. Nine of the 13 teachers who were teaching pattern drills at that 
time accepted an invitation to participate in a half-day session in which 
they would derive the basic design. They were randomly assigned to 
one of two discussion groups, in addition to the teachers each group 
included two administrators or supervisors from the central office 
who were actively concerned with the planning and instructional aspects of 
the program. 

A day or two before the meeting the evaluator oriented two discussion 
leaders--both members of the Office of Re search- -briefing them on the 
background of the program and mapping out a general procedure for 
conducting the groups and recording the proceedings. It was decided to 
use the circular response technique, in which each participant has a chance 
to make a statement in his turn; the reason for this strategy being that no 
one can dominate the entire discussion and that each individual has an 
opportunity to get his thinking into the record. 
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On the day of the meeting the evaluator appeared to assume a 
secondary role. He opened the meeting and greeted the participants, 
attempting to dispel some of the anxiety that seems to be in the air 
when people who don’t know one another too well are getting ready to 
take part in a new kind of a.ctivity. He introduced the staff from the 
Office of Research, outlined the overall plan for the morning’s work, 
and identified the discussion leaders. He then directed the participants 
to their assigned tables. From that moment until almost the very end 
of the meeting he receded into the background and became a careful 
observer --on call in case of an emergency or breakdown in the 
discussion, but otherwise a silent partner in the enterprise. 

A description of the discussion between groups and the interaction 
of participants is enlightening. Despite the random assignment of indi- 
viduals, each group took on its own unique characteristics. One of 
the groups was much more open than the other, with each participant 
willing to take his turn, eager to comment and respond to the leader’s 
non-directive guidance. The other group seemed noticeably less secure. 
New teachers shied away from making comments whenever possible, 
and looked to their more experienced colleagues to give them clues. Even 
more noticeable was the difference made by the presence in this group 
of the program director. Although not known as an authoritarian figure, 
she was obviously perceived as the ultimate decision maker. Even the 
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oicU'.r teachers in this group tended to look to her for significant 
cues as to her intent, and were generally content to echo her pronounce- 
ments. Thanks to the ability of each of the discussion leaders, the 
data of each of these groups proved to be reasonably similar. 

The evaluator’s role now became that of a recorder of consensus. 

By categorizing the responses and fitting them into the broad dimensions 
of the Output, Input, and Process elements of the ’’Program Design 

Criteria, ” he developed the formal program design. Every member of 
the Pattern Drills staff received a copy of this design as formal feedback. 

Appendix "A" contains a copy of this report. 

The next step in the evaluation process was to hold a panel meeting 
so that the adequacy of the design could be assessed. A linguistics 
consultant served as the technical expert on the panel. The program 
director and her supervisor represented the program staff. The coordi- 
nator of evaluation and the evaluator, joined by a staff psychologist, made 
up the evaluation component of the panel meeting. For several hours 
this team put each statement in the program design under the micioscope. 
The evaluator had several responsibilities at the panel meeting. Besides 
setting up the meeting, he answered questions concerning evaluation of 
the program and fed into the record a summary of the data on the compat- 
ibility of the program with the entire school system that he had elicited 
from teachers in school visitations prior to the meeting. 
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Before distributing the report of the panel proceedings, the program 
evaluator conferred with the program director, explaining the findings 
in an effort to prepare her for the discrepancy report. With her approval, 
the proceedings were sent to the entire program staff with an accompanying 
letter of explanation. This distribution constituted formal feedback. The 
report is shown in Appendix "B. !l 

We had now completed a full cycle of evaluation, all of which per- 
tained to Stage I activity. Even though, according to the panel, the 
program was not yet adequately designed, and thus further Stage I 
work v/as indicated, the evaluation panel had raised several questions 
about operations feasibility and had supplemented the design sufficiently 
to provide a standard for measuring operation. We were now ready 
for a new cycle of evaluation, which would examine Stage II concerns 
while at the same time tightening up some Stage I definitions. 

The principal source for identifying specific areas of performance 
to be studied was the program design itself as standard, which the eval- 
uator examined item by item. However, in the panel meeting, there 
had been agreement that "Outputs" needed clarification. This in turn 

raised the question as to whether teachers understood the objectives of 
pattern drills instruction. Similarly, it v/as clear that time constraints 
were a probable source of difficulty in the field. In this connection it 
was noted that the time dimension was closely related to staff functions 
and duties. 
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An ad hoc interview instrument for collecting data on these areas 
of program performance was designed. See Appendix "C. " A random 
sample of pattern drills teachers was drawn, and interviews were 
conducted in the schools by the evaluator and an assistant from the 
Office of Res'earch. Analysis of the data revealed the following findings: 
First, a discrepancy did exist between desired and actual performance 
in the time devoted to pattern drills instruction. Second, a majority 
of teachers thought that the drills should not be taught as often as 
the consultant had recommended. Teachers explained the discrepancy 
between wish and fulfillment by stating that other curriculum demands 
left insufficient time for pattern drills, and that many students * especially 
in the upper grades, found the content too juvenile to engage their 
serious attention. Third, teachers were found to be lacking in their 
under standing of the program's objectives, with 39 percent mistaking 
inappropriate objectives for legitimate ones. 

These findings were reported both formally and ini ’-.nally to the 
program director. She was willing to take action regarding the 
confusion over valid and invalid objectives. It is believed that this 
willingness was one of the factors that led to a stepped-up in-service 
program for all pattern drills teachers. She questioned the concern 
over time constraints, disagreeing with the consultant's recommendation 
that the drills should be taught for at least 10 minutes every day. This 
difference of opinion, although somewhat ameliorated recently, is still 




short of complete resolution. The report containing these findings wa3 
distributed to all members of the program staff as is shown in 
*t "D. " 

hr.\p.-.y it in the Stage II findings of Standard Speech Development was 
d ?' j further design the "process" of pattern drills instruction. To 
a -jcs&pKsh this, the evaluator drew up a flow chart to show how the 
teaching of the grammar drills on the one hand and the phonetic drills 
on the other v/as designed to lead to students' control of standard English 
speech. See Appendix "E. " In this chart the relationships of objectives 
at various levels of specificity are indicated. This is an instance of 
Stage II work pointing the way to further Stage I activity in the area of 
making the program increasingly more specific. 

Influenced by Stage II findings, the' program director instituted the 
previously mentioned in-service program for all teachers. A skilled 
instructor was sent into each school to conduct a half-day workshop in 
methods and objectives. Her goal was to equip teachers to handle 
pattern drills with greater understanding and effectiveness. Citing her 
own experience, she stated that reality did not permit a daily lesson 
of 10 to 15 minutes. She suggested instead a minimum of three 10-m.inute 
lessons weekly. 

The in-service program became the focal point for the next cycle of 
evaluation, which would attempt to measure the effect of the training on 
classroom practice. Substantially the same instrument was administered 



3.S was employed for tne previous evaluation, and the same procedure 
was used fco collect the data. Although respondents were asked virtually 
the same questions as before, the rationale underlying the data 
collection had a somewhat different emphasis. The previous in ter- 
views were conducted to determine the degree of initial correspondence 
between design and teacher performance in the field. The second set 
of observations was designed to examine the effectiveness of an 
intervening staff training activity on two key program variables — under- 
standing of objectives and time constraints. Indirectly it also provided 
a measure of the effect of earlier feedback on program operation. 

The following findings were noted: The discrepancy between g ui de - 
lines and practice in time allocation had not been reduced in the period 
between the two observations. These findings as reported are shown in 
Appendix "F. 11 

Teachers, not surprisingly in view of the instructor's conception of 
time requirements, still did not see the need for daily presentation of.the 
drills. Nor v/ere they realizing the reduced time allotment they them- 
selves thought desirable. There was, however, a slight improvement 
in their understanding of objectives, but not enough to conclude that this 
was no longer a serious departure from design specifications. The 
report which provided feedback on this series of evaluation activities 
listed several alternatives that might improve understanding of objectives 



&nd resolve the time conflict. This report has been discussed with the 






program director, and has been distributed to department chairmen in 
the schools as well as other key members of the staff. 

Two significant modifications have already been made. First, the 
program has been markedly contracted. It will no longer be offered 
in grade nine. As of January 196 9 it will be mandated in a maximum 



m 



of four schools, chosen by the program director on the basis of the 
availability of skilled instructors and supervision by department 



chairmen. Second, federal funding has been withdrawn from the program, 
at least partly because of the discrepancies noted in evaluation reports. 

To summarize, several significant curriculum and evaluation 
developments have taken place side by side as the Standard Speech 
Development Program has evolved. Y *.oi r» the program point of view, 
to date these developments have been. 

1. Proposing the program 

2. Writing the curriculum 

3. Training teachers 

- 4. Installing the program 

5. Withdrawing federal funding and contracting the program 

The following evaluation activities have paralleled and influenced the 
program developments listed above: 

1. Deriving the program design 
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2. Assessing the program design * • 

3. Comparing field operation with design 

4. Designing instruments to measure pupil achievement 

5. Providing continuous feedback to program staff in ac- 
cordance with provisions of the evaluation model 

This year, 1968-69, Stage III work will begin. The actual 
application of continuous assessment in the evaluation cf a spe- 
cific program raises specific problems in the area of measure- 
ment. Before the appropriate measures can be determined, 
several definitions are needed: 

1. Operational definitions of treatment variables 

2. Operational definitions of dependent variables 

3. A definition of the intervals of time between measure- 
ments 

The formulation of all three sets of definitions is the respon- 
sibility of the program staff. 

In the case of the Standard Speech Development Program., 
the actual lessons are controlled by a script which the teacher 
reads. As a function of the amount of time he spends on the 
lesson, he will use all or only some of the examples, which are 
all equivalent. Thus, the measurable treatment variables be- 
come amount of time and/or number of examples used. The 
teachers will keep a record of these two variables for each lesson 



3-nd these will be averaged over the period between any two measurement 
of the dependent variable. 

The operational behaviors for the dependent variables are: 

1. Given a set of sentences, the student can select the 
sentence which is in standard speech form. 

2. Given a part of a sentence, the student can orally pro- 
duce the complete sentence in standard form. 

The forms are such that it is possible to generate any 
number of equivalent sentences. An achievement test can be 
constructed testing these two behavioral objectives. Using 
enough examples, it is possible to divide the achievement test 
into a number of mini- achievement tests--each of which is 
equivalent. These can then be used to plot the achievement 
growth during the year, either separately for each objective, 
or grouped for both objectives. These tests would be adminis- 
tered at those intervals defined by the program staff. This 
method permits the continuous assessment of the program dur- 
ing the year, and permits a more precise analysis of the effect 
of the treatment variables on the dependent variables. 

The day of reckoning, represented by Stage J.V evaluation, 
lies somewhere in the future. Accordingly, the evaluation staff 
has been considering adequate measurement of student terminal 
behavior. A search of the literature and conferences with con- 
sultants made it clear that we would need tt> develop our own 



instruments for the Standard Speech Development Program. Appendix 
"G" shows one of two instruments under development. It was determined 
that we would ultimately need to know whether students who had received 
the treatment could control standard speech and the subsidiary enabling 
skills on which that control is based. A staff psychologist assigned to 
the Office of Research has been given the major responsibility for design- 
ing and testing the- required instruments. It is hoped that validity infor- 
mation for these instruments will be available during the present year. 

This brings us to the next cycle of evaluation for the program which 
will look back to Stages I and II and ahead to Stages III and IV during the 



current sciiool year. 



APPENDIX A 



PATTERN DRILLS PROGRAM DEFINITION 

General 

I. Overall Statement of Objectives and Rationale for the Program 

The principal objective of the Pattern Drills Program is to provide 
adolescents who ordinarily speak non-standard English in all situa- 
tions with the ability to speak the standard English of Western 
Pennsylvania when the occasion calls for its use. The rationale for 
the program acknowledges the place of both non-standard and 
standard speech. 

II. Scope 

A. Number of Pupils and Schools Involved 

At the end of the 1966-1967 school year, the program served 
approximately 5, 100 students in 20 qualifying secondary schools. 

B. The Grades or Ages of Participants 

Students served by the program include all those enrolled in 
grades 7 and 8 in participating schools. 

C. General Description of Staff 

The staff for the Pattern Drills Program is made up of all • 
teachers of English in grades 7 and 8 in participating schools. 
Supervision is provided by the Supervisor of English regulairly 
assigned to the schools involved. 






(Outcome s 



I. Major Objectives- -the changes that are expected to take place in 
program participants as a result of their experiences in the pro- 
gram. There are two types of major objectives. 

A. Terminal Objectives — as a direct result of the Pattern Drills 

Program, it is expected that students will have the following 
skills: 

1. Be able to communicate clearly with all speakers of English 

2. Be able to shift automatically from non-standard to standard 
speech and vice-versa as the situation requires 

B. Ultimate Objectives--those things which it is expected tha.t the 

Pattern Drills Program will contribute to its participants in the 
long run: 

1. Increased job opportunities 

2. Increased self-confidence 

3. Increased opportunity for participation in the activities of 
middle-class society 

4. Increased enthusiasm for participation and achievement in 
English classes 

5. Increased ability and willingness to communicate with 
speakers of standard English 

II. Enabling Objectives- -m order to bring about the major objectives 
listed above, the student must first accomplish several things 
through the program: 

A. Be aware of the importance of standard speech in appropriate 
situations 

B. Respect the appropriateness of non-standard dialects in specific 
circumstances 

C. Be able to produce the sounds and syntax of standard spoken 
speech 
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D. Be able to imitate different patterns of standard English 

E. Be able to hear and distinguish between standard English and 
non-standard dialects 

III. Other Benefits--benefits expected to accrue to the community as a 

result of the Pattern. Drills Program: 

A. A general upgrading of the community as its citizens are able to 
participate increasingly in economic and social activities brought 
about in part by newly acquired control of middle-class speech 

B. A gradual elimination of non-standard speech as today’s non- 
standard speakers extend their knowledge and use of standa.rd 
English 



Antecedents 



I. Students 

A. Selection Criteria 

The Pattern Drills Program was in effect in two of twenty 
qualifying secondary schools from May 1 through the end of the 
1966-1967 school year. All seventh- and eighth-grade students 
in these schools participated, the total number coming to 1250. 
The only prerequisite mentioned for the program, aside from 
being enrolled in either the seventh or eighth grade in these 
schools, was "an understanding of English vocabulary. " This 
sole requirement points up two significant observations: 

1. A principal difference between the use of pattern drills in 
foreign language and standard English instruction lies in the 
fact that in learning a foreign language the student must be 
taught to receive as well as transmit the patterns; but in 
learning standard English the non-standard speaker already 
has a passive understanding of the patterns to be mastered. 

2. Hence, in the present program, total energies can be focused 
on giving students control of phonological and grammatical 
patterns with which they are already at least passively fami- 
liar. This means that it is not generally necessary to avoid 
the use of lexical items for fear that they would be unknown 
to the children. This observation supports the consultant’s 
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previous finding in analyzing the tapes of students' speech 
that lexical items were "so minimal a$ io be negligible. " 

B. Entering Behaviors 

The students involved in the Pattern Drills Program, though far 
from a homogeneous group, have in common many observable 
characteristics which must be taken into consideration when 
planning instructional activities: 

1. A majority of the students entering the program cannot con- 
trol standard English. 

2. Many students come from homes in which standard English 
is neither spoken nor accepted. 

3. A large number of students feel that they would be ridiculed 
if they were to use standard English in thsir community. 

4. Some students resist standard English because, in the opin- 
ions of others, they fear that its acquisition wiil lead adults 
to expect too much of them. 

5. Many students expect language instruction to offer them a 
practical tool for communication. 

II. Staff 

The most important persons in the Pattern Drills Program are the 
individual classroom teachers, who must have as basic qualifications 
the ability to speak standard English and at least minimal knowledge 
of the purposes and techniques of pattern drills. In addition, they 
should be enthusiastic and convey a lack of prejudice concerning 
dialect differences. 

III. Support 

A. Administrative Support 

Teachers look to the principal (and at Westinghouse to the depart- 
ment chairman) to provide the day-to-day support for the program 
within a school, such as scheduling pattern drills classes to the 
language laboratory. As for overall city-wide support, the cen- 
tral office staff is expected to provide the materials, funds, and 
communication necessary to initiate and. maintain a successful 
program. 
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B. Human Resources — the following persons* services are im- 
portant to program implementation: 

1. The linguistics consultant has the following major roles: 

a. To develop and explain the philosophy of pattern 
drills instruction 

b. To identify the patterns of standard and non-standard 
speech which are to form the content of the pattern 
drills 

Cc To help the pattern drills writing committee with 
the production of the drills 

d. To demonstrate the techniques of teaching the drills 

e. To provide analysis and feedback to pattern drills 
teachers 

2. The instructional leader of English at Westinghouse co- 

ordinates the program with the larger English curricu- 
lum in the school. 

3. Other teachers can facilitate the objectives of the Pattern 
Drills Program by stressing the same structures and pro- 
nunciations that are covered in the formal drills. 

C. Media- -the four most valuable materials and items of equip- 
ment and their purposes are the following: 

1. The pattern drills , which provide the actual instructional 
content for the program and assure that a particular 
pattern is correctly presented with respect to rhythm, 
continuity, and purity 

2. Charts prepared by the Office of Research and the pat- 
tern drills writing committee, which are used for mo- 
tivation and visual cues 

3. A tape recorder so that students may hear and evaluate 
their speech 

4. The language laboratory , which effectively aids develop- 
ment of oral language skills 
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Process 



I. Studer?.t Activities 

The drills prepared for the present program are based on a careful 
comparison between the grammatical and phonological patterns of 
the non-standard and standard varieties of English spoken in the 
Pittsburgh area because it is in this region that the vast majority 
of the students will live and work. The very nature of pattern drills, 
which utilize the aural-oral techniques also employed in modern 
foreign language instruction, leads to two basic student activities: 

1. Listening to the standard English sound or grammatical form 

2. Repeating the bandar d sound or grammatical form in a variety 

of drill practices in large groups, small groups, and individually 

Several observations were made concerning the second of the two 
basic activities listed above: 

a. Each separate drill must be limited to a specific sound or 
grammatical form. 

b. In order to reinforce and provide for eventual automatic 
control of the standard pattern, frequent substitution drills 
are presented in which students concentrate on nonessential 
substitutions in phrase or sentence content while they are 
repeating the desired pattern unchanged. ^ 

c. Occasional drills are designed for testing, but the main 
activity for students revolves around using the drills for 

pattern practice, reflecting the major objectives of the 
program. 

For example, in a drill devoted to the standard use of "he doesn’t " 
the students might repeat the following series of sentences, each time" 
focusing their attention on the changing direct object of the verb, while 
the pattern the teacher wishes to reinforce (“he doesn’t") remains con- 
stant and seemingly of secondary significance: 



He doesn’t see the elephant. 

He doesn’t see the giraffe. 

He doesn’t see the tiger. 

He doesn’t see the hippopotamus, 
etc. 



II. 



Staff Functions and Activities — the specified functions and specific 
duties of the pattern drills teacher are the following: 



Duties 



Functions 



1. Teaches pattern drills 



2, Plans for coordinating 

pattern drills with the total 
English curriculum 



3. Evaluates student progress 

4. Serves on writing committee 
if appointed 

5. Communicates with others 
regarding pattern drills 
experience 



a. Motivates student for drills 
(Method varies with individual 
drills, teacher, and class) 

b. Presents drills and guides 
responses by use of oral and 
visual cues 

Allots time for drills within 
the total English curriculum 
b. Incorporates knowledge and 
skills into rest of English 
program 

Conducts test drills 

Produces drills for classroom 
use 

Provides feedback to writing 
committee 



B. Intra-staff Communication and Co or dination- -the following intra- 
staff activities provide for communication about and coordination 
of pattern drills: 

I. At ”We 3 tinghou 8 e, teachers are kept informed of developments 
by the instructional leader of English and the department chair- 
man. 

2* There is informal contact among teachers of pattern drills. 

3. Meetings are held between teachers and the Associate Director 
of Instruction for English and the English supervisor. 

4. In-service sessions are conducted in the schools and at the 
Administration Building by the associate director., the English 
supervisor, and the linguistics consultant. 
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standard to standard speech 
when the situations requires 
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teaching the drills 
To provide analysis and feedback to 
pattern drills teachers 
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APPENDIX C 



PATTERN DRILLS 
CYCLE II INTERVIEW SCHEDULE 

In the continuing development of the Pattern Drills Program it is 
desirable to determine the viewpoints of teachers at periodic intervals. 

With this in mind, we are requesting your appraisal of Pattern Drills 
at this time in terms of your experience with them in your own class- 
room. The Office of Research guarantees the anonymity of all respondents. 

PART ONE - TIME DIMENSION 

1. How many times each week do you feel Pattern Drills should 
be presented in: 

Grade 6 Grade 7 Grade 8 Grade 9 ? 

2. How many times each week do you ordinarily teach Pattern 
Drills in: 

Grade 6 Grade 7 Grade 8 Grade 9 ? 

3. If there is a difference between your answers to Questions 
1 and 2, to what do you attribute the discrepancy? 



4. How much time do you feel should be devoted to each Pattern 
Drills session in: 

Grade 6 Grade 7 Grade 8 Grade 9 ? 

" * - - 11 ' 1 i . — tm 

5. How much time do you ordinarily devote to each session in: 

Grade 6 Grade 7 Grade 8 Grade 9 ? 

6. If there is a difference between your answers to Questions 
4 and 5, to what do you attribute the discrepancy? 



/ 2 2 / 8 9 - 



PART TWO - OBJECTIVES OF PATTERN DRILLS 



A. Which, of the u >jectives listed below do you feel genuinely apply 
to Pattern Drills? Indicate your opinion by placing a check mark before 
those objectives you believe pertain to the program. Please mark the 
check in the first of the two blank spaces that precede the item: 

As a result of participation in Pattern Drills instruction, students 
should better be able: 

To eliminate most gross errors in written composition 

To communicate clearly with all English-speaking 

persons with whom they come in contact 

To generalize to standard speech forms in contexts 

other than those presented in the formal drills 

To substitute formal acceptable words and phrases for 

overused slang expressions 

To spot errors in pronunciation and grammar in the 

language of their friends 

To use appropriate speech patterns automatically 

To increase their formal vocabulary 

To reproduce the sounds and grammatical constructions 

of standard English 

To achieve success in the study of a foreign language 

To differentiate between situations for which standard 

or non-standard speech is appropriate 

To shift from non-standard speech and vice versa as 

the situation requires 

To instruct their parents and other adults in correct 

usage 

To speak standard English in all situations 

To overcome noticeable speech impediments not 

requiring the services of a speech therapist 







B, Now indicate the importance you assign to the objectives you 
listed as legitimate ones for Pattern Drills instruction by rank-ordering 
those you have checked. Start with number 1 for the most important^ 
and continue until you have recorded a number for all the objectives in 
this category. Write your figures in the second of the two blanks pre- 
ceding the objective. 
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RATIONALE FOR 

PATTERN DRILL PROGRAM INTERVIEW SCHEDULE 



The rationale for administering the attached pattern drills instru^ 
ment is quoted from the program’s gycle II plan, dated January 11, 1968: 



PROGRAM QUESTION RATIONALE 

DIMENSIONS 



Staff Functions 
and Duties 


How consistently and 
uniformly are pattern 
drills being taught? 

Are all teachers using 
pattern drills? 


. . . The panel stated that 
staff duties and functions 
were not specified in the 
definition. The first 
step in specifying duties 
and functions of the 
teacher is to determine 
what he /she is doing in 
the classroom. 


Major Objectives 
and Enabling 
Objectives 


Are teachers aware 
of (a) major objec- 
tive s ? 


If teachers are to pre- 
sent drills properly, 
they should be able to 
(a) state program’s 
overall objectives. . . 



It should be further noted that at a panel meeting on May 3, 1967 
the linguistics consultant stated that in order for students to achieve 
automatic control of standard speech (a key terminal objective), pat- 
tern drills should be taught for at least 15 minutes a day. Part one 
of the instrument will determine to what extent this recommendation 
is currently being honored. 

The data will be tabulated so that separate analyses can be made 
for elementary school a.ud secondary school respondents. 
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PROCEDURES FOR ADMINISTERING 
PATTERN DRILLS PROGRAM INTERVIEW SCHEDULE 



Before beginning the interview, the interviewer will either ask 
the respondent to read the introductory paragraph or he will orally 
explain the purpose of the interview, making the same points as are 
contained in the opening paragraph. 

Part One- -Time Dimension 



1. The questions on this page will constitute an oral interview, 
with the interviewer recording responses as they are obtained. 

2. Any discrepancies noted between questions 1 and 2 should be 
summarized in brief anecdotal form in the space below question 3, and 
any discrepancies noted between questions 4 and 5 should be similarly 
summarized in the space below question 6. If no discrepancies are 
noted, write "None" or “No discrepancy. 11 

3. At the top of the page the interviewer will record the teacher 1 
name and school together with the amount of in-service training the 
teacher has had for pattern drills to the nearest half day. 

Part Two- -Objectives of Pattern Drills 

1. This part of the instrument is a questionnaire, which is to be 
completed by the respondent in the interviewer's presence. 

2. In the first column to the left of the page the respondent will 
check all the objectives which he considers valid for pattern drills. 

3. In the second column the respondent will rank-order the five 
objectives which he considers to be the five most important, with #1 
indicating the highest priority. 
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APPENDIX D 



CYCLE II REPORT 
PATTERN DRILLS PROGRAM 

An evaluation report of the Pattern Drills Program was issued in 
September 1967. The findings indicated that the program was, in general, 
compatible with the English program and the overall school program, al- 
though there was confusion on the part of teachers as to the relative pri- 
orities in the use of class time. The definition of the Pattern Drills 
Program v/as found inadequate in four areas. (For an elaboration of 
these findings the reader is referred to the previous document. Evaluation 
Report, Stage I, Pattern Drills Program. ) Investigation by the evaluation 
staff has produced no evidence of action by the program staff as a re- 
sponse to these findings. 

The current second cycle of evaluation took place during the first 
semester of this school year. The study was undertaken to determine 
the degree to which the operating Pattern Drills Program is consistent 
with the specifications of the program definition. 

Program Operation 

The evaluation focused on two questions: (1) How are teachers 
using the drills, i. e. , how often are the drills taught and for what 
length of time at each session? and (2) Do teachers understand the 
purposes of pattern drills instruction? The findings are presented 
under separate headings below. 






Use of the Drills* Using two points of reference, it was found 



that a discrepancy exists between desired and actual performance in 
the use of the drills. When actual performance was compared with 
what teachers themselves perceived as desirable, a substantial number 
were found who were neither using the drills as frequently as desired 
nor holding drill sessions of the desired length. When actual per- 
formance was compared with the standards recommended by the pro- 
gram consultant, the vast majority of teachers were found not to be 
performing in accordance with the desired standards. 

Purposes of the Program. In general, teachers do not understand 
the purposes of pattern drills instruction. When they were asked to 
select "correct" and "incorrect" items from a list of possible objec- 
tives for the program, 39 percent of their responses were inappropriate. 
Many of the "incorrect" choices were statements in direct conflict with 
the purposes of the program. The significance of this misunderstand- 
ing for the effectivenrs of the program is pointed up in the following 
statement by the program consultant: "If teachers are to present drills 
properly, they should be able to. . . state the program's overall ob- 
jectives. . . " 

Changes Effected in the Program 

Investigation by the evaluation staff has shown that action has 
already been taken consistent with the findings of the cycle II evaluation. 





Beginning in February 1968, a consultant will spend a half day in 
junior high schools conducting appropriate in-service activities with 
pattern drills teachers. 



-99 





APPENDIX E 



CO 

CD 

U 

o 

o 



vu 



2 

p 

2 

Q 



2 

ci 

W 

H 

H 

g 



O 



a 

5 

a 

Q 

a 

Ui 

H 

P 

6 
S 

W 

CD 




u 

Q 



rt 

O 



C£ 

O 



o 

c 

o 

J2 

Ch 



CsJ 



p 

o 

H- f 

tc 




t> □ £>□ 



)<►□ “Q O D 



/o/ J/oj/-ivi-]/otf 



?A 







appendix f 



STANDARD SPEECH DEVELOPMENT (PATTERN DRILLS) 

CYCLE HI REPORT 

Summary of Preceding R e port 

The most recent evaluation report of the Standard Speech Develop- 
ment Program, ^ the Cycle H Report , was distributed to staff members 
in February 1968. It was concerned with two major questions: 

(1) How often were pattern drills being taught and for what length of 
time at each session, and (2) Whether teachers understood the program's 
objectives. Data collected in January 1968 showed pronounced discrep- 
ancies between the time recommended for teaching pattern drills 
(15-minute sessions per day) and time actually devoted to them in many 
classrooms (ranging from 0-5 sessions per week of 0-10 minutes per 
session). The report also highlighted a confusion in teachers' minds 
concerning the valid objectives of the program. This inability of many 
teachers to distinguish between valid and inappropriate objectives cast 
considerable doubt upon the program's successful implementation. 

Scope of Present Report 

The present document reports the impact of a recent in-service 
training which attempted to change teachers' understanding of the time 

■^Formerly called Pattern Drills Program 
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requirements and of the program's objectives. This in-service pro- 
gram was conducted after the issuance of the Cycle II Rep ort. It was 
conducted in most of the participating schools, and included the major- 
ity of teachers in the program. According to the instructor, the 
training focused on the practical day-to-day operation of the program. 
In order to facilitate comparison, the same group of randomly selected 
teachers who were interviewed for the previous study in January were 
again polled, being asked to answer the same questions. In addition, 
these teachers were asked for an appraisal of the recent in-service 
training in terms of its value in enabling them to help students to 
realize the program's objectives. 

Time Allocated to Drills 

The serious discrepancy between guidelines and actual practice 
in the time allocated to the teaching of pattern drills has not been 
reduced since the preceding report. In fact, the discrepancy may 
have been accentuated by the in-service activity. In May 1967 the 
program's consultant recommended that pattern drills be taught for 
a minimum of 15 minutes every day in order to provide students with 
automatic control of standard speech. She reinforced this recommen- 
dation in a memorandum to the Office Of Research in February 1968> 
by stating: 
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... I recommend the use of the pattern drills materials 
for a minimum of 1 5 minutes five days a week. . . The 
regular daily use of drills cannot be emphasized too 
strongly, since the immediate goal of developing a new 
set of language habits depends heavily on regular and 
repeated practice to establish automatic control. 

Nevertheless, confronted with practical considerations as she perceiv- 
ed them, the in-service instructor advised teachers to schedule no 
more than three five- to- ten minute pattern drills lessons weekly. The 
following findings of the current evaluation underline the conflict 
between expert opinion and classroom practice regarding time 
requirements: 

1. Teachers do not see the need for daily presentation of the drills. 

2. They fall somewhat short of realizing even the reduced 
frequency of presentation that they consider desirable. 

3. They devote less time to teaching the drills in each 
succeeding grade. 

4. Teachers are more likely to approach the recommended 
number of lessons per week than the recommended length 
per session. 

5. The two most common reasons which teachers gave to 

explain the time discrepancy were: (a) Too crowded a 

curriculum, and (b) A lack of student interest. 



Teachers’ Unde rstanding of Program Objectives 

A slight improvement was noted in teachers’ ability to identify 
the valid objectives of the program (69 percent appropriate responses 
versus 61 percent in the previous interviews). 
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However, half of the teachers still saw two spurious objectives as 
valid: (1) To substitute formal acceptable words and phrases for 

overused slang expressions, and (2) To overcome noticeable speech 
impediments not requiring the services of a speech therapist. 

Further, one-third of the teachers identified two other invalid objec- 
tives as authentic: (1) To spot errors in the pronunciation and grammar 

in the language of their friends, and (2) To increase their formal 
vocabulary. These findings indicate no substantial change in under- 
standing of objectives as a result of the in-service training. 

Teachers' Appraisal of In-service Training 

Over half the teachers credited the current in-service activity 
with helping them to lead their students to realize two crucial 
program objectives: (1) The ability to reproduce the phonology and 

grammar of standard English, and (2) The power to use appropriate 
speech patterns automatically. However, the continued uncertainty 
of many teachers regarding program objectives is pointed up by the 
fact that almost one- fourth also stated that the recent in-service 
training has helped them to accomplish invalid objectives. 

Problem Analysis 

Several basic problems have characterized the Standard Speech 
Development Program since its inception. These have been further 
clarified by this cycle of evaluation. 
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.1. Time Allocated to Drills. In May 1967 the program's consultant 



i'ecommended that pattern drills be taught for a minimum of 15 
minutes per day. However, the in-service instructor advised 
teachers to schedule no more than three five- to- ten minute pattern 
drills lessons per week. In practice, teachers are unable or unwill- 
ing to schedule pattern drills lessons as frequently as recommended. 
This problem could be ameliorated by: 

1. Realigning the components of the present course of study 
in English in participating schools to make a definite daily 
pattern drills lesson possible. 

2. Substituting the pattern drills program for large portions 

of the sections on grammar and speech in the present course 
of study. The three-track construction of the present 
English curriculum may make it feasible to effect this sub- 
stitution for the lower track classes with comparative ease. 

3. Modifying objectives by specifying a more limited, less 
rigorous set of expectations, which would in turn reduce the 
amount of time needed for the program. 

2. Teachers' Understanding of Program Objective s. The data reflect 
that teachers still do not have a complete understanding of the 
program’s objectives. The seriousness of this is emphasized by the 
consultant 5 s statement that: "If teachers are to present drills 
properly they should be able to . . . state the program's overall 
objectives. . . " 

3. Motivation of Students. There is a need to provide teachers with 
techniques to improve the motivation of students. This need becomes 
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increasingly imperative in the upper grades, to judge from the 
teachers 1 observations that the decreasing amount of time devoted 
to pattern drills in grades eight and nine is partly due to lack of student 



interest. 

Both improved understanding of objectives and greater competence 
in motivating and presenting pattern drills may be accomplished by: 

1. Intensified in-service training of sufficient duration 
be±ore teachers begin to teach drills. This training should 
be designed to acquaint teachers with the philosophy of 
pattern drills as they relate to participating students in 
the Pittsburgh Public Schools. 

2. An ongoing in-service program during the school year, using 
demonstration, observation, and supervision tailored to 
specific classrooms and grade levels. 

3. The revision of existing materials to increase the relevance 
of the content for participating students. Attention should be 
given to writing new drills to attract the more mature 
students in the upper grades or to lowering the grade place- 
ment for the present program. 

• 4. The preparation of new materials concerning objectives and 
techniques for teacher reference. 

A detailed report of the data and analysis can be found in the 
forthcoming annual report. Future evaluation will reexamine those 
aspects of the program considered here and will study the effect of 
pattern drills instruction on pupil performance as instruments to 
measure achievement now fcsing developed become available. 
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PRELIMINARY TEST 

Aural Discrimination Between Standard and Non-Standard 
English Sentences. 

Narrator: You will hea.- a statement followed by two comparison 

statements. On your answer sheets circle the letter 
A or B identifying the statement which is most like 
the first in terms of language structure. 



Example 1: I ain't got none. 

Comparison A: I don't have any. 

Comparison B: I ain't got none. 

Now circle your answer. (5 second pause) 
The correct answer is B. Do you see why? 
If not raise your hand. 

Here is another example. 

Example 2: Dis is worser dan dat. 

Comparison A: Dis is worser dan dat. 

Comparison B: This is worse than that. 

Now circle your answer. (5 second pause) 
The correct answer is A. Do you see why? 
If not raise your hand. 

Now we will begin. 



Number 1: Jane look all right in dat dress. 

A: Jane looks all right in that dress. 

*B: Jane look all right in dat dress. 

2: Jim's shirt don't look clean. 

A: Jim's shirt doesn't look clean. 

#B: Jim's shirt don't look clean. 

3: The children rided on de bus. 

A: The children rode on the bus. 

❖B: The children rided on de bus. 



H3 . 





4 : 



5 : 



6 : 



7 : 



8 : 



9 : 



10 : 



11 : 



12 : 



13 : 



14 : 



John an Susan haven' went ta church yet . 

A: John and Susan haven't gone to church yet. 

*B: John an Susan haven* went ta church yet. 

At las, de TV man fix de set. 

*A: At las, de TV man fix de set. 

B: At last, the TV man fixed the set. 



Mary may have come in for the interviews. 

A: Mary may have came in for de interviews. 

*B: Mary may have come in for the interviews. 



I got no money. 

*A: I got no money. 

B: I don't have' any money. 

He looks like a policeman. 

A: He's look like a policeman. 

*B: He looks like a policeman. 



It happen again. My car was broke down. 

'•'A: It happen again. My car was broke down. 

B: It happened again. My car broke down. 



I waited until they got back, but June and Tom were late. 
V A: I waited until they got back, but June and Tom 

were late. 

B: I wait until they got back, but June an Tom was 

late. 



I try to dance as best I could. 

*A: I try to dance as best I could. 

B: I try to dance as well as I can. 



The party was not fun. I don't think anything happened. 
*A: The party was not fun. I don't think anything 

happened. 

B: De party was no fun. I don' think nothin happen. 

Bob! It looks like the dog catcher is coming out. 

A: Bob! It look like de dogcatcher's comin out. 

'•'B: Bob! It looks like the dogcatcher is coming out. 

He try to acts like de boss. 

*A: He try to acts like de boss. 

B: He tries to act like the boss. 
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15; This is a nice car. 

*A: This is a nice car. 

B: Dis here's a nice car. 

Look- -there are airplanes in the sky. 

A: Look--Der's airplanes in da sky. 

❖B: Look- -there are airplanes in the sky. 

After they win de ballgame they was so happy. 

A: After they won the ballgame they were so happy. 

❖B: After they win de ballgame they were so happy. 

We were talking to some of the teachers. 

A: We was talkin to some of de teachers. 

#B: We were talking to some of the teachers. 

But maybe some of they are just sad. 

A: But maybe some of dem just sad. 

^B: But maybe some of them are just sad. 

That's just the name of the song. 

A: That jes de name of de song. 

❖B: That's just the name of the song. 

21; She bout 16, I guess. 

A: She's about 16, I guess. 

*B: She bout 16, I guess. 

22: Are you gonna have a party on Saturday? 

A: Are you going to have a party on Saturday? 

*B: Are you gonna have a party on Saturday? 

23: I baby-sitted and made fifty cent. 

*A: I baby-sitted and made fifty cent. 

B: I baby-sat and made fifty cents. 

24: When I broked my leg it hurted alot. 

❖A: When I broked my leg it hurted alot. 

B: When I broke my leg it hurt a lot. 

25: They were so happy we came to the basketball game. 

#A: They were so happy we came to the basketball game. 

B: They was so happy we come to de basketball game. 



16: 



17: 
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19: 
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There was three of us at de store. 

*A: There was three of us at de store. 

B: There were three of us at the store. 

I think they in classroom now. 

A: I think they're in the classroom now. 

*B: I think they in classroom now. 

She in 2nd grade, but I's in 4th. 

A: She is in 2nd grade, but I'm in 4th. 

*B: She in 2nd grade, but I's in 4th. 

They're waiting for the icecream man. 

#A: They're waiting for the icecream man. 

B: They waitin for de icecream man. 

After eatin too much, usually I be sick. 

❖A: After eatin too much, usually I be sick. 

B: After eating too much, usually I am sick. 

I am hungry. I hope there is a candy machine. 

A: I is hungry. I hope there be a candy machine. 

*B: I am hungry. I hope there is a candy machine. 

If John don't want trouble, they be there. 

A: If John doesn't want trouble, they will be there. 

*B: If John don't want trouble, they be there. 

I like hotdogs cause dey taste good. 

A: I like hotdogs because they taste good. 

*B: I like hotdogs cause dey taste good. 

We will be traveling when it's summer. 

A: We be traveling when it's summer. 

❖B: We will be traveling when it's summer. 

If I were you, I would have Jack pay for the scraped fender. 
A: If I's you, I be having Jack pay for the scraped 

fender. 

❖ B: If I were you, I would have Jack pay for the 

scraped fender. 

Some of the kids are riding in the Ford. 

A: Some of the kids be ridin in de Ford. 

❖B: Some of the kids are riding in the Ford. 




37: Dat stove don’t be electric. 

A: That stove isn’t electric. 

❖B: Dat stove don’t be electric. 

38: What’s the matter? Everytime I come looking for her, 

she isn’t at home. 

A: What's da matter? Everytime I come lookin for 

her, she don't be at home. 

*B: What’s the matter? Everytime I come looking for 

he r , she isn't at home. 

39: No, I don't want any lnnch. I'm not hungry now! 

A: No, I don' want no lunch. I don' be hungry now! 

❖B: No, I don't want any lunch. I'm not hungry now! 

40: We are going to the zoo. 

A: We' s goin to the zoo. 

*B: We are going to the zoo. 

41: My brudder be plavin in de yard. 

❖A: My brudder be playin in de yard. 

B: My brother is playing in the yard. 

42: Der's a fire in de kitchen! 

❖ A : Der's a fire in de kitchen! 

B: There’s a fire in the kitchen! 

43: Ain't ya never on time? 

❖A: Ain't ya never on time? 

B: Aren't you ever on time? 

44: Hey Tom, what are you doing there? 

❖A: Hey Tom, what are you doing there? 

B: Hey Tom, what chew doin der? 

45: He isn’t a mechanic. My car still doesn't run. 

❖A: He isn't a mechanic. My car still doesn’t run. 

B: He ain't no mechanic. My car still don't run. 

46: Sorry mister, I don't know where those stores are. 

A: Sorry mista, I don' know where dos stores is. 

*B: Sorry mister, I don't know where those stores 

are. 
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47: No Jack, we aren't going to the grocery store. 

*A: No Jack, we aren't going to the grocery store. 

B: No Jack, we ain't goin' to de grocery store. 

48: We ain't never had such a good dance! 

*A: We ain't never had such a good dance! 

B: We have never had such a good dance! 

49: I da know. Maybe tha's what she done. 

A: I don't know. Maybe that's what she did. 

*B: I da know. Maybe tha's what she done. 
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50: I've got to learn how to swim. 

A: I gotta learn ta swim. 

*B: I've got to learn how to swim. 

51: Ya' seen our ballfield? 

*A: Ya' seen our ballfield? 

B: Have you seen our ballfield? 

52: I did'n do it! I neva been der! 

A: I didn't do it. I've never been there! 

❖ B: I did'n do it! I neva been der! 

53: Teacha, I ain't got no pencil. 

*A: Teacha, I ain't got no pencil. 

B: Miss Smith, I don't have a pencil. 

54: We're going to talk about that movie. 

* A: We gonna talk about dat movie. 

*B: We're going to talk about that movie. 

55: ’ Hey Bill, what do you mean by that? 

#A: Hey Bill, what do you mean by that? 

B: Hey Bill, what ya mean by dat? 

56: All you're worrying about is the money. 

❖A: All you're worrying about is the money. 

B: All you worrin 1 bout is de money. 

57: We ain't talkin', Miss. Ain't nobody said nothin'. 

A: We aren't talking, Miss Jones. Nobody said 

anything . 

*B: We ain't talkin'. Miss. Ain't nobody said nothin'. 
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58: When you late, you be in trouble. 

*A: When you late, you be in trouble. 

B: When you're late, you're in trouble. 

59: What da ya think? It look like an old Plymouth. 

A: What do you think? It looks like an old Plymouth. 

*B: What da ya think? It look like an old Plymouth. 

60: I don 4 t like it. But ask Allen how he feels about it. 

A: I don' like it. But ast Allen how do he feel 'bout 

it. 

#B: I don't like it. But ask Allen how he feels about 

it. 

61: Pizza and spaghetti is alright, but Jane like hotdogs instead. 

A: Pizza and spaghetti are all right, but Jane likes 

hotdogs instead. 

*B: Pizza and spaghetti is alright, but Jane like 

hotdogs instead. 

62: You watch it] Your brain is going to break out of your 

he ad I 

A: You watch it] You' brain gonna bust out you' 

head] 

*B: You watch itl Your brain is going to break out of 

your head] 

63: I asked my mother if I could go, but she said no. 

*A: I asked my mother if I could go, but she said no. 

B: I ast my mother could I go, but she says no. 

64: We chase the dog out de house. 

*A: We chase the dog out de house. 

B: We chased the dog out of the house. 

65: You mean in school you be restless? 

A: Do you mean that in school you are restless? 

*B: You mean in school you be restless? 

66: This TY show is worse than that one. 

*A: This TV show is worse than that one. 

B: Dis TV show is worser dan dat one. 



o 
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67! What chew doin’? Kin I help? 

A: What are you doing? Can I help? 

*B: What chew doin'? Kin I help? 

68: Ail them football players is real good I 

A: All of those football players are good! 

^B: All them football players is real good! 

69: I ast my girl ta go to the dance. 

*A: I ast my girl ta go to the dance. 

B: I asked my girl to go to the dance. 

70: She's washing all of those dishes. 

*A: She's washing all of those dishes. 

B: She washin' all them dishes. 
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